Neural networks occupy a large branch of research in machine intelligence. Artificial neural networks are information-processing devices inspired by the interconnected, parallel structure of animal brains. They take the form of software or hardware networks having collections of mathematical models that emulate some of the observed characteristics of nervous systems and analogies to adaptive biological learning. Generally, they are composed of large numbers of interconnected processing elements, which can be realized in software or hardware, that are analogous to the neurons of an animal brain. The connections between these processing elements are weighted in a fashion believed to be analogous to synapses.
Training a neural network involves making adjustments to the “synaptic” connections that exist between the neurons (i.e., the values of the weights). Training is performed by exposure to a set of input/output data where the training algorithm iteratively adjusts the connection weights. These connection weights store the knowledge necessary to solve specific problems.
Neural networks are being applied to greater and greater numbers of practical problems, including very complex ones. They are particularly suited to pattern recognition and classification problems having many inputs such as speech recognition, character and signal recognition, and functional prediction and system modeling where the physical processes are not understood or are highly complex.
There are many types of neural networks. Some of the more popular include the multilayer perceptron, which is generally trained with the backpropagation of error algorithm, learning vector quantization, radial basis function, Hopfield, and SOFM. Some are classified as feedforward and others as recurrent (i.e., implement feedback) depending on how data is processed through the network. Another feature of neural networks is the mechanism by which it is trained. Some use a technique called supervised training while others are referred to as unsupervised or self-organizing. In supervised training, the network is guided by an instruction process while in unsupervised algorithms the data is clustered into similar groups based on the attributes that provide the inputs to the algorithms.
The SOFM or Kohonen artificial neural network is a type of unsupervised neural network. In unsupervised learning, an untrained neural network is exposed to examples or input vectors and internal parameters adjusted. In SOFMs, all the neurons of the network receive the same input. The nodes engage in competition with their neighbors and at each stage of a self-learning process, the one with most activity “wins.” Learning is based on the concept of winner neurons.
Unsupervised learning allows the objects to be grouped together on the basis of their perceived closeness in n-dimensional hyperspace (where n is the number of variables or observations made on each object). Such methods, then, although in some sense quantitative, are better seen as qualitative since their chief purpose is merely to distinguish objects or populations.
Referring to FIG. 1, SOFMs provide an objective way of classifying data through self-organizing networks of artificial neurons. There are two layers, an input layer 110 and a competition layer 100. Each node of the input layer may be connected (as indicated by connectors 120) to the entire set of nodes in the competition layer. In an example configuration, each neuron may be connected to its eight nearest neighbors on a grid. The neurons store a set of weights (a weight vector) each of which corresponds to one of the inputs in the data. The objective of a Kohonen network is to map input vectors (patterns) of arbitrary dimension N onto a discrete map lying in a competition layer of arbitrary dimension, but typically of 1 or 2 dimensions. The algorithm adjusts weights so that patterns close to one another in the input space should be close to one another in the map: they should be topologically ordered.
The learning process is as follows: First, the weights for each output unit are initialized, typically to random starting values. An iterative process is performed that end when weight changes are negligible. For each of a number of input patterns a winning output node and all units in the neighborhood of the winner are identified and the weight vectors for all of these nodes updated. The winning output unit is simply the unit with the weight vector that has the smallest Euclidean distance to the input pattern. The neighborhood of a unit is defined as all units within some distance of that unit on the map (not in weight space). If the size of the neighborhood is 1 then all units no more than 1, either horizontally or vertically, from any unit fall within its neighborhood. The weights of every unit in the neighborhood of the winning unit (including the winning unit itself) are updated such that each unit in the neighborhood is moved closer to the input pattern. As the iterations are performed, the learning rate is reduced. If the parameters are well chosen the final network should capture the natural clusters in the input data.
The factor that governs the size of the weight alterations is known as the learning rate. The adjustments to each item in the weight vector are made in accordance with the following:ΔWi,j=α(Ij−Wi,j)sin d/(2d)where WIj is the jth weight of the ith node, α is the learning rate, Ij is the jth component of the input vector, and d is the distance between the current node and the winner. The above formula is one of a number that are known in the prior art and the invention to be discussed in subsequent section could make use of it or any other. As mentioned, the process of training continues until the changes in the weights falls below some predetermined value in successive iterations.
The effect of the “learning rule” (weight update algorithm) is to distribute the neurons evenly throughout the region of n-dimensional space populated by the training set. The neuron with the weight vector closest to a given input pattern will win for that pattern and for any other input patterns that it is closest to. Input patterns which allow the same node to win are then deemed to be in the same group, and when a map of their relationship is drawn a line encloses them. In the resulting map, it is possible to examine closely relationships between the items in the training set and visualize these relationships even for complex structures in high-dimensional input spaces.
There are two phases to the process of generating a solution. In an initial first stage, in the prior art, the learning rate begins at a high value close to unity and is gradually monotonically decreased. The rate of decrease may be exponential, linear, or some other and according to the prior art, the particular pattern has not generally been regarded as particularly important. During the initial phase, called the “ordering phase” the topological ordering of the weight vectors takes place. A long convergence phase follows and is associated with fine-tuning of the map. The learning rate is maintained at low values (well below 0.1, for example) and monotonically and progressively decreased with each iteration.
The quality of a SOFM solution for any given problem is by no means assured. Where the input vectors are high-dimensional and complex, results may take a very long time to converge and may even produce poor end results.